Outlier Detection Using SemiDiscrete Decomposition

Semidiscrete decomposition (SDD) is usually presented as a storage-eecient analogue of singular value decomposition. We show, however, that SDD actually works in a completely diierent way, and is best thought of as a bump-hunting technique; it is extremely eeective at nding outlier clusters in datasets. We suggest that SDD's success in text retrieval applications such as latent semantic indexing is fortuitous, and occurs because such datasets typically contain a large number of small clusters. Abstract: Semidiscrete decomposition (SDD) is usually presented as a storage-eecient analogue of singular value decomposition. We show, however, that SDD actually works in a completely diierent way, and is best thought of as a bump-hunting technique; it is extremely eeective at nding outlier clusters in datasets. We suggest that SDD's success in text retrieval applications such as latent semantic indexing is fortuitous, and occurs because such datasets typically contain a large number of small clusters.